Storage networks can comprise several Fibre Channel (FC) switches interconnected in a fabric topology. These switches are interconnected by a number of inter-switch links (ISLs), which carry both data and control information. An ISL is terminated at a port on each of the two switches it connects to. The ISL typically provides a physical link between the two switches. The basic information carrier in Fibre Channel is a frame. Thus when data is sent via a Fibre Channel network, the data is generally encapsulated within a frame. Frames can be transmitted between switch ports over the ISL.
To prevent a target device (either host or storage) from having to drop frames, the Fibre Channel architecture provides flow control mechanisms based on a system of credits. Each of these credits represents the ability of the device to accept additional frames. If a recipient issues no credits to the sender, no frames can be sent. Pacing the transport of subsequent frames on the basis of this credit system helps prevent the loss of frames and reduces the frequency of entire Fibre Channel sequences needing to be retransmitted across the link.
Upon arrival at a receiver, a frame goes through several steps. It is received, deserialized, decoded, and stored in a receive buffer where it is processed by the receiving port. If another frame arrives while the receiver is processing the first frame, a second receive buffer is needed to hold this new frame. Unless the receiver is capable of processing frames as fast as the transmitter is capable of sending them, it is possible for all of the receive buffers to fill up with received frames. When that happens, if the transmitter sends another frame, the receiver will not have a receive buffer available and the frame may be lost.
The flow control mechanism of the Fibre Channel architecture prevents this type of error condition by allowing the receiver to control when the transmitter may send frames. This flow control mechanism is referred to as buffer to buffer credit. Credits are the number of frames a buffer can hold per fabric port. The number of credits may be different for each port, and even for virtual channels of each port. The number of available credits for each port is generally exchanged during the login process between two ports. When a Fibre Channel HBA (host bus adapter) is connected to a Fibre Channel switch, buffer to buffer credits (BB credits) are exchanged between the HBA and the switch port during login using FLOGI and FLOGI Response. In the FLOGI and FLOGI_RSP payloads, the Common Service Parameters contains a field for BB credit which indicates the number of buffers available for the port.
Buffer to buffer credit control flow works by a sending port using its available credit supply and waiting to have the credits replenished by the port on the opposite end of the link. The credit value is decremented when a frame is sent and replenished when a response is received. If the available credits for a given port reaches zero, the supply of credits is said to be exhausted. Further transmission of frames with that port is then suspended until the amount of credits can be replenished to a non-zero value. The rate of frame transmission is generally regulated by the receiving port in part based on the availability of buffers to hold received frames.
This buffer to buffer credit mechanism works effectively when credits are sent and received correctly. However, BB credits are sometimes lost during their transmission. BB credits may become lost, for example, when the credit is corrupted in transit, when a bad optical link causes errors in SOF (Start Of Frame) resulting in the remote port not seeing the frame and not returning the credit, or when a misbehaving remote port does not respond to the frames being sent by the local port. The depletion in number of BB Credits that were initially exchanged between the two ports during login causes the ports to operate with fewer buffer credits. This results in performance degradation as lost credits negatively impact throughput. Because lost credits are generally not restored until a link reset or link offline event is triggered, performance does not go back to normal until such an event occurs. To prevent this type of performance degradation as a result of lost BB credits, the Fibre Channel Standard provides a mechanism for recovery of lost BB credits.
The credit recovery mechanism of the Fibre Channel standard, specifically in FC-FS3, section 19.4.9, though very useful in restoring the performance of the system, is not efficient and can be improved upon. As Fibre Channel systems become more complex and operate in higher bandwidth environments, credit recovery and proper credit allocation become more essential for overall system performance. Therefore, what is required is a process and system that efficiently and effectively detects and recovers lost credit.
Summary
In one embodiment, a buffer to buffer credit recovery mechanism is disclosed. Buffer to buffer credit recovery involves exchanging primitive control signals and parameters during the login sequence to enable credit recovery. Once credit is lost; there may be a need for resetting a link to reset the credit recovery counters and BB credits. Both of these processes require synchronization between the ports involved in the credit recovery mechanism. This synchronization is achieved by enabling credit recovery during the Link Reset protocol negotiation and ensuring that no frames or R_RDYs are exchanged during the procedure.